fix(reranker): detect pre-normalized scores and use rank-based normalization#1512
Merged
nicoloboschi merged 2 commits intoMay 25, 2026
Merged
Conversation
…ization External API rerankers (SiliconFlow, Cohere, etc.) return pre-normalized relevance_score in [0, 1] with very small absolute values. Applying sigmoid to these compresses everything to ~0.5, destroying the ranking signal and making recency the sole sorting factor. This fix detects the score range: - If all scores are in [0, 1]: use rank-based normalization with tie handling (equal scores get equal ranks) - Otherwise (logits): use sigmoid as before This preserves the correct behavior for local models (logits) while fixing ranking quality for external API rerankers.
nicoloboschi
reviewed
May 7, 2026
nicoloboschi
left a comment
nicoloboschi
left a comment
Collaborator
There was a problem hiding this comment.
lgtm, can you add unit tests on this function
- Rank-based normalization for [0,1] scores - Tied scores receive identical normalized values - Sigmoid normalization for logit scores - Empty candidates returns [] without calling predict() - Fix typo: "sole排序 factor" -> "sole sorting factor"
Contributor
Author
|
Unit tests added for
Also fixed a typo in the comment: "sole排序 factor" → "sole sorting factor". Tests use AsyncMock, no external API calls. |
nicoloboschi
approved these changes
May 25, 2026
r0gig0r
added a commit
to r0gig0r/hindsight
that referenced
this pull request
May 26, 2026
Notable upstream additions pulled in: - feat(api): clear endpoint for mental model content (vectorize-io#1706) - feat(api): per-operation LLM concurrency caps (vectorize-io#1738) - feat(typescript-client): concrete generated types (replace Promise<any>) - feat(reranker): Alibaba Qwen3-Rerank support (vectorize-io#1501) - feat: opencode-go LLM provider (vectorize-io#1652) - feat(extensions): OperationValidator.precheck pre-body-parse hook (vectorize-io#1548) - feat(right-agent): new Right Agent integration (vectorize-io#1599) - fix(ollama): ollama-cloud provider + native API auth (vectorize-io#1734) - fix(reflect): hide disabled tools from agent system prompt (vectorize-io#1740) - fix(retain): split oversized single items in batch retain (vectorize-io#1736) - fix: escape literal braces in user-supplied prompt fields (vectorize-io#1728) - fix(mental-models): full refresh pending delta baselines (vectorize-io#1684) - fix(api): lazy load reflect tiktoken encoding (vectorize-io#1654) - fix(api): reject blank retain content (vectorize-io#1685) - fix(api): auto-refresh openai-codex OAuth access_token (vectorize-io#1637) - fix(api): gzip middleware for graph payloads (vectorize-io#1731) - fix(reranker): detect pre-normalized scores; rank-based fallback (vectorize-io#1512) Conflicts: only package-lock.json files (took upstream, npm install verified) Fork customizations verified intact (all 14 checks): - duplicate_checker_fn streaming Phase 1.5 in orchestrator - FallbackLLMProvider + CircuitBreaker (fallback_llm.py) - Single-fact consolidation mode (is_fallback_active routing) - recallExp + Jaccard dedup + compact memory formatter (plugin) - Codex 5.1-codex-mini reasoning guard - Infinity reranker /models fallback in cross_encoder.py - diversity.py + deduplication.py fork-only modules retained Tests: - openclaw vitest: 267/267 pass - ruff: clean - tsc --noEmit: clean - pytest: pre-existing env-config flakes (need HINDSIGHT_API_LLM_API_KEY); upstream commit 90cb145 acknowledged as pre-existing CI flakes Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
External API rerankers (SiliconFlow, Cohere, etc.) return pre-normalized
relevance_scorein[0, 1]with very small absolute values. The current code applies sigmoid to all scores, assuming they are logits. This compresses everything to~0.5, destroying the ranking signal and making recency the sole sorting factor.Example with SiliconFlow BAAI/bge-reranker-v2-m3
With sigmoid, all scores are
~0.5and recency becomes the only ranking signal. With rank-based normalization, the CE signal correctly dominates.Fix
Detect the score range in
CrossEncoderReranker.rerank():[0, 1]: Use rank-based normalization with tie handling (equal scores get equal ranks). This preserves relative ordering without depending on absolute score magnitudes.cross-encoder/ms-marco-MiniLM-L-6-v2).Testing
Verified with real SiliconFlow API scores:
Unit tests added in
tests/test_reranker_score_normalization.py.Related